On May 31 last year, 25-year-old Safarain Herring was shot in the head and dropped off at St. Bernard Hospital in Chicago by a man named Michael Williams. He died two days later.
Chicago police eventually arrested the 64-year-old Williams and charged him with murder (Williams maintains that Herring was hit in a drive-by shooting). A key piece of evidence in the case is video surveillance footage showing Williams’ car stopped on the 6300 block of South Stony Island Avenue at 11:46 p.m.—the time and location where police say they know Herring was shot.
Videos by VICE
How did they know that’s where the shooting happened? Police said ShotSpotter, a surveillance system that uses hidden microphone sensors to detect the sound and location of gunshots, generated an alert for that time and place.
Except that’s not entirely true, according to a motion filed by Williams’ public defender.
That night, 19 ShotSpotter sensors detected a percussive sound at 11:46 p.m. and determined the location to be 5700 South Lake Shore Drive—a mile away from the site where prosecutors say Williams committed the murder, according to a motion filed by Williams’ public defender. The company’s algorithms initially classified the sound as a firework. That weekend had seen widespread protests in Chicago in response to George Floyd’s murder, and some of those protesting lit fireworks. (While the address listed on the real-time alert was about a mile away from where the shooting allegedly occurred, the alert also contained a map and GPS coordinates identifying the intersection where the shooting allegedly occurred.)
But after the 11:46 p.m. alert came in, a ShotSpotter analyst manually overrode the algorithms and “reclassified” the sound as a gunshot. Later, the company issued a forensic report with a map and GPS coordinates identifying the same intersection that ShotSpotter had initially identified in the real-time alert but updated the address to be closer to the actual GPS coordinates it initially identified.
“Through this human-involved method, the ShotSpotter output in this case was dramatically transformed from data that did not support criminal charges of any kind to data that now forms the centerpiece of the prosecution’s murder case against Mr. Williams,” the public defender wrote in the motion.
The document is what’s known as a Frye motion—a request for a judge to examine and rule on whether a particular forensic method is scientifically valid enough to be entered as evidence. Rather than defend ShotSpotter’s technology and its employees’ actions in a Frye hearing, the prosecutors withdrew all ShotSpotter evidence against Williams.
The case isn’t an anomaly, and the pattern it represents could have huge ramifications for ShotSpotter in Chicago, where the technology generates an average of 21,000 alerts each year. The technology is also currently in use in more than 100 cities.
Motherboard’s review of court documents from the Williams case and other trials in Chicago and New York State, including testimony from ShotSpotter’s favored expert witness, suggests that the company’s analysts frequently modify alerts at the request of police departments—some of which appear to be grasping for evidence that supports their narrative of events.
Untested evidence
Had the Cook County State’s Attorney’s office not withdrawn the evidence in the Williams case, it would likely have become the first time an Illinois court formally examined the science and source code behind ShotSpotter, Jonathan Manes, an attorney at the MacArthur Justice Center, told Motherboard.
“Rather than defend the evidence, [prosecutors] just ran away from it,” he said. “Right now, nobody outside of ShotSpotter has ever been able to look under the hood and audit this technology. We wouldn’t let forensic crime labs use a DNA test that hadn’t been vetted and audited.”
Sam Klepper, senior vice president for marketing and product strategy at ShotSpotter, told Motherboard in an email that the company has no reason to believe the prosecutor’s decision reflects a lack of faith in its technology.
ShotSpotter evidence and employee testimony has been admitted in 190 court cases, he wrote. “Whether ShotSpotter evidence is relevant to a case is a matter left to the discretion of a prosecutor and counsel for a defendant… ShotSpotter has no reason to believe that these decisions are based on a judgment about the ShotSpotter technology,” he said.
The Chicago Police Department, Cook County State’s Attorney’s Office, Mayor Lori Lightfoot’s office, and Alderman Chris Taliaferro, who chairs the city council’s public safety committee, did not respond to interview requests or questions.
A pattern of alterations
In 2016, Rochester, New York, police looking for a suspicious vehicle stopped the wrong car and shot the passenger, Silvon Simmons, in the back three times. They charged him with firing first at officers.
The only evidence against Simmons came from ShotSpotter. Initially, the company’s sensors didn’t detect any gunshots, and the algorithms ruled that the sounds came from helicopter rotors. After Rochester police contacted ShotSpotter, an analyst ruled that there had been four gunshots—the number of times police fired at Simmons, missing once.
Paul Greene, ShotSpotter’s expert witness and an employee of the company, testified at Simmons’ trial that “subsequently he was asked by the Rochester Police Department to essentially search and see if there were more shots fired than ShotSpotter picked up,” according to a civil lawsuit Simmons has filed against the city and the company. Greene found a fifth shot, despite there being no physical evidence at the scene that Simmons had fired. Rochester police had also refused his multiple requests for them to test his hands and clothing for gunshot residue.
The original ShotSpotter audio files of the fifth shot were deleted, leaving a “redact[ed]” copy of the recording with the five alleged gunshots.
Both the company and the Rochester Police Department “lost, deleted and/or destroyed the spool and/or other information containing sounds pertaining to the officer-involved shooting,” according to Simmons’ civil suit. “Greene acknowledged at plaintiff’s criminal trial that employees of ShotSpotter and law enforcement customers with an audio editor can alter any audio file that’s not been locked or encrypted.”
A jury ultimately acquitted Simmons of attempted murder and a judge overturned his conviction for possession of a gun, citing ShotSpotter’s unreliability.
Greene—who has testified as a government witness in dozens of criminal trials—was involved in another altered report in Chicago, in 2018, when Ernesto Godinez, then 27, was charged with shooting a federal agent in the city.
The evidence against him included a report from ShotSpotter stating that seven shots had been fired at the scene, including five from the vicinity of a doorway where video surveillance showed Godinez to be standing and near where shell casings were later found. The video surveillance did not show any muzzle flashes from the doorway, and the shell casings could not be matched to the bullets that hit the agent, according to court records.
During the trial, Greene testified under cross-examination that the initial ShotSpotter alert only indicated two gunshots (those fired by an officer in response to the original shooting). But after Chicago police contacted ShotSpotter, Greene re-analyzed the audio files.
“An hour or so after the incident occurred, we were contacted by Chicago PD and asked to search for—essentially, search for additional audio clips. And this does happen on a semi-regular basis with all of our customers,” Greene told the court, according to a transcript of the trial. He later ruled that there were five additional gunshots that the company’s algorithms did not pick up.
Greene also acknowledged at trial that “we freely admit that anything and everything in the environment can affect location and detection accuracy.”
ShotSpotter analysts “agree with the machine classification over 90% of the time,” Klepper, from ShotSpotter, wrote to Motherboard. “In a tiny number of cases, our customers request us to perform a location analysis to validate the accuracy of the location. If we find an error, we provide a more accurate location to the customer to assist the investigation.”
Prior to the trial, the judge ruled that Godinez could not contest ShotSpotter’s accuracy or Greene’s qualifications as an expert witness. Godinez has appealed the conviction, in large part due to that ruling.
“The reliability of their technology has never been challenged in court and nobody is doing anything about it,” Gal Pissetzky, Godinez’s attorney, told Motherboard. “Chicago is paying millions of dollars for their technology and then, in a way, preventing anybody from challenging it.”
The evidence
At the core of the opposition to ShotSpotter is the lack of empirical evidence that it works—in terms of both its sensor accuracy and the system’s overall effect on gun crime.
The company has not allowed any independent testing of its algorithms, and there’s evidence that the claims it makes in marketing materials about accuracy may not be entirely scientific.
Over the years, ShotSpotter’s claims about its accuracy have increased, from 80 percent accurate to 90 percent accurate to 97 percent accurate. According to Greene, those numbers aren’t actually calculated by engineers, though.
“Our guarantee was put together by our sales and marketing department, not our engineers,” Greene told a San Francisco court in 2017. “We need to give them [customers] a number … We have to tell them something. … It’s not perfect. The dot on the map is simply a starting point.”
In May, the MacArthur Justice Center analyzed ShotSpotter data and found that over a 21-month period 89 percent of the alerts the technology generated in Chicago led to no evidence of a gun crime and 86 percent of the alerts led to no evidence a crime had been committed at all.
Klepper disputed those findings to Motherboard, saying that “the data source used to draw their conclusions, on its own, results in an incomplete picture of an incident” because a gun may have been fired even if there is no documented police evidence that it was.
He also said that Greene’s testimony in the San Francisco trial “had nothing to do with the determination of our actual historical accuracy rate. While marketing and sales have appropriate input on our service level guarantees for our contracts, actual accuracy rates are based on detections that we record.”
Meanwhile, a growing body of research suggests that ShotSpotter has not led to any decrease in gun crime in cities where it’s deployed, and several customers have dropped the company, citing too many false alarms and the lack of return on investment.
One recent study of ShotSpotter in St. Louis found that ShotSpotter “has little deterrent impact on gun-related violent crime in St. Louis. [Automated gun detection systems] also do not provide consistent reductions in police response time, nor aid substantially in producing actionable results.”
Klepper contested those and other research findings, saying that “the studies’ conclusions do not reflect what we see.”
He pointed to a 2021 study by New York University School of Law’s Policing Project that determined that assaults (which include some gun crime) decreased by 30 percent in some districts in St. Louis County after ShotSpotter was installed. The study authors disclosed that ShotSpotter has been providing the Policing Project unrestricted funding since 2018, that ShotSpotter’s CEO sits on the Policing Project’s advisory board, and that ShotSpotter has previously compensated Policing Project researchers.
Chicago pushes back
Chicago is one of the most important cities in ShotSpotter’s portfolio and is increasingly becoming a battleground over its use.
If a court ever agrees to examine the forensic viability of ShotSpotter, or if prosecutors continue to drop the evidence when challenged, it could have massive ramifications. From January 2017 through June 2021, ShotSpotter reported 94,313 gunfire incidents in the city, an average of 20,958 per year, according to data obtained by Motherboard through a public records request.
Chicago is ShotSpotter’s second biggest client, after New York City, accounting for 13 percent of the company’s revenue during the first quarter of 2021. But Chicago’s $33 million contract with the company is coming to an end and city officials must decide this August whether or not to renew it.
Meanwhile, the city is grappling with new research, a rise in shootings, cases like the Williams and Godinez trials, and tragedies that have prompted renewed criticism of the technology.
It was a ShotSpotter alert in the early-morning hours of March 29 that dispatched police to a street in Little Village where they eventually shot and killed 13-year-old Adam Toledo, who was unarmed at the time.
That and other recent events have sparked a new campaign by community and civil rights groups in Chicago calling on city officials to drop ShotSpotter.
“These tools are sending more police into Black and Latinx neighborhoods,” Alyx Goodwin, a Chicago organizer with the Action Center on Race and the Economy, one of the groups leading the campaign, told Motherboard. “Every ShotSpotter alert is putting Black and Latinx people at risk of interactions with police. That’s what happened to Adam Toledo.”’
Motherboard recently obtained data demonstrating the stark racial disparity in how Chicago has deployed ShotSpotter. The sensors have been placed almost exclusively in predominantly Black and brown communities, while the white enclaves in the north and northwest of the city have no sensors at all, despite Chicago police data that shows gun crime is spread throughout the city.
Community members say they’ve seen little benefit from the technology in the form of less gun violence—the number of shootings in 2021 is on pace to be the highest in four years—or better interactions with police officers.
“If you had relationships with any of the people on the block, you wouldn’t need the technology, ’cause we could tell you,” Asiaha Butler, president of the Resident Association of Greater Englewood, told Motherboard. Instead, the technology seems to have given police another excuse not to build relationships with residents. When shots ring out in the neighborhood, police may respond faster, but it’s an “over-militarized police presence. You see a lot of them. It’s not a friendly interaction,” she said.
Editor’s Note: Following the publication of this article, VICE received copies of court documents from the Michael Williams case, which show that ShotSpotter did not change the coordinates of the gunfire by a mile, but had identified the same intersection for the gunfire in both its initial real-time alert and in its later detailed forensic report. The article has also been updated to clarify that the original recording of the gunshots in the Silvon Simmons case were deleted, but that the jury heard a redacted copy of the recording with the five alleged gunshots.
